<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Disparity map post-filtering</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d3/d81/tutorial_contrib_root.html">Tutorials for contrib modules</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Disparity map post-filtering </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h2>Introduction </h2>
<p>Stereo matching algorithms, especially highly-optimized ones that are intended for real-time processing on CPU, tend to make quite a few errors on challenging sequences. These errors are usually concentrated in uniform texture-less areas, half-occlusions and regions near depth discontinuities. One way of dealing with stereo-matching errors is to use various techniques of detecting potentially inaccurate disparity values and invalidate them, therefore making the disparity map semi-sparse. Several such techniques are already implemented in the StereoBM and StereoSGBM algorithms. Another way would be to use some kind of filtering procedure to align the disparity map edges with those of the source image and to propagate the disparity values from high- to low-confidence regions like half-occlusions. Recent advances in edge-aware filtering have enabled performing such post-filtering under the constraints of real-time processing on CPU.</p>
<p>In this tutorial you will learn how to use the disparity map post-filtering to improve the results of StereoBM and StereoSGBM algorithms.</p>
<h2>Source Stereoscopic Image </h2>
<div class="image">
<img src="../../ambush_5_left.jpg" alt="ambush_5_left.jpg"/>
<div class="caption">
Left view</div></div>
<div class="image">
<img src="../../ambush_5_right.jpg" alt="ambush_5_right.jpg"/>
<div class="caption">
Right view</div></div>
 <h2>Source Code </h2>
<p>We will be using snippets from the example application, that can be downloaded <a href="https://github.com/opencv/opencv_contrib/blob/master/modules/ximgproc/samples/disparity_filtering.cpp">here</a>.</p>
<h2>Explanation </h2>
<p>The provided example has several options that yield different trade-offs between the speed and the quality of the resulting disparity map. Both the speed and the quality are measured if the user has provided the ground-truth disparity map. In this tutorial we will take a detailed look at the default pipeline, that was designed to provide the best possible quality under the constraints of real-time processing on CPU.</p>
<ol type="1">
<li><b>Load left and right views</b> <div class="fragment"><div class="line">    Mat left  = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">imread</a>(left_im ,<a class="code" href="../../d8/d6a/group__imgcodecs__flags.html#gga61d9b0126a3e57d9277ac48327799c80af660544735200cbe942eea09232eb822">IMREAD_COLOR</a>);</div><div class="line">    <span class="keywordflow">if</span> ( left.empty() )</div><div class="line">    {</div><div class="line">        cout&lt;&lt;<span class="stringliteral">&quot;Cannot read image file: &quot;</span>&lt;&lt;left_im;</div><div class="line">        <span class="keywordflow">return</span> -1;</div><div class="line">    }</div><div class="line"></div><div class="line">    Mat right = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">imread</a>(right_im,<a class="code" href="../../d8/d6a/group__imgcodecs__flags.html#gga61d9b0126a3e57d9277ac48327799c80af660544735200cbe942eea09232eb822">IMREAD_COLOR</a>);</div><div class="line">    <span class="keywordflow">if</span> ( right.empty() )</div><div class="line">    {</div><div class="line">        cout&lt;&lt;<span class="stringliteral">&quot;Cannot read image file: &quot;</span>&lt;&lt;right_im;</div><div class="line">        <span class="keywordflow">return</span> -1;</div><div class="line">    }</div></div><!-- fragment --> We start by loading the source stereopair. For this tutorial we will take a somewhat challenging example from the MPI-Sintel dataset with a lot of texture-less regions.</li>
<li><b>Prepare the views for matching</b> <div class="fragment"><div class="line">            max_disp/=2;</div><div class="line">            <span class="keywordflow">if</span>(max_disp%16!=0)</div><div class="line">                max_disp += 16-(max_disp%16);</div><div class="line">            <a class="code" href="../../d5/df1/group__imgproc__hal__functions.html#ga2fe39d2201b12e1b961ca56b2aff9ff2">resize</a>(left ,left_for_matcher ,<a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(),0.5,0.5, <a class="code" href="../../da/d54/group__imgproc__transform.html#gga5bb5a1fea74ea38e1a5445ca803ff121ac00f4a8155563cdc23437fc0959da935">INTER_LINEAR_EXACT</a>);</div><div class="line">            <a class="code" href="../../d5/df1/group__imgproc__hal__functions.html#ga2fe39d2201b12e1b961ca56b2aff9ff2">resize</a>(right,right_for_matcher,<a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(),0.5,0.5, <a class="code" href="../../da/d54/group__imgproc__transform.html#gga5bb5a1fea74ea38e1a5445ca803ff121ac00f4a8155563cdc23437fc0959da935">INTER_LINEAR_EXACT</a>);</div></div><!-- fragment --> We perform downscaling of the views to speed-up the matching stage at the cost of minor quality degradation. To get the best possible quality downscaling should be avoided.</li>
<li><b>Perform matching and create the filter instance</b> <div class="fragment"><div class="line">            Ptr&lt;StereoBM&gt; left_matcher = StereoBM::create(max_disp,wsize);</div><div class="line">            wls_filter = <a class="code" href="../../da/d17/group__ximgproc__filters.html#ga8a351f67b897bb7cdaccaef115bafcac">createDisparityWLSFilter</a>(left_matcher);</div><div class="line">            Ptr&lt;StereoMatcher&gt; right_matcher = <a class="code" href="../../da/d17/group__ximgproc__filters.html#gae85b598094c2d747f84f22b2aa2c2e0c">createRightMatcher</a>(left_matcher);</div><div class="line"></div><div class="line">            <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#ga397ae87e1288a81d2363b61574eb8cab">cvtColor</a>(left_for_matcher,  left_for_matcher,  <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#gga4e0972be5de079fed4e3a10e24ef5ef0a353a4b8db9040165db4dacb5bcefb6ea">COLOR_BGR2GRAY</a>);</div><div class="line">            <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#ga397ae87e1288a81d2363b61574eb8cab">cvtColor</a>(right_for_matcher, right_for_matcher, <a class="code" href="../../d8/d01/group__imgproc__color__conversions.html#gga4e0972be5de079fed4e3a10e24ef5ef0a353a4b8db9040165db4dacb5bcefb6ea">COLOR_BGR2GRAY</a>);</div><div class="line"></div><div class="line">            matching_time = (double)<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>();</div><div class="line">            left_matcher-&gt; compute(left_for_matcher, right_for_matcher,left_disp);</div><div class="line">            right_matcher-&gt;compute(right_for_matcher,left_for_matcher, right_disp);</div><div class="line">            matching_time = ((double)<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>() - matching_time)/<a class="code" href="../../db/de0/group__core__utils.html#ga705441a9ef01f47acdc55d87fbe5090c">getTickFrequency</a>();</div></div><!-- fragment --> We are using StereoBM for faster processing. If speed is not critical, though, StereoSGBM would provide better quality. The filter instance is created by providing the StereoMatcher instance that we intend to use. Another matcher instance is returned by the createRightMatcher function. These two matcher instances are then used to compute disparity maps both for the left and right views, that are required by the filter.</li>
<li><b>Perform filtering</b> <div class="fragment"><div class="line">        wls_filter-&gt;setLambda(lambda);</div><div class="line">        wls_filter-&gt;setSigmaColor(sigma);</div><div class="line">        filtering_time = (double)<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>();</div><div class="line">        wls_filter-&gt;filter(left_disp,left,filtered_disp,right_disp);</div><div class="line">        filtering_time = ((double)<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>() - filtering_time)/<a class="code" href="../../db/de0/group__core__utils.html#ga705441a9ef01f47acdc55d87fbe5090c">getTickFrequency</a>();</div></div><!-- fragment --> Disparity maps computed by the respective matcher instances, as well as the source left view are passed to the filter. Note that we are using the original non-downscaled view to guide the filtering process. The disparity map is automatically upscaled in an edge-aware fashion to match the original view resolution. The result is stored in filtered_disp.</li>
<li><b>Visualize the disparity maps</b> <div class="fragment"><div class="line">        Mat raw_disp_vis;</div><div class="line">        <a class="code" href="../../da/d17/group__ximgproc__filters.html#gae192a87924cbc134b75e01b79bb44718">getDisparityVis</a>(left_disp,raw_disp_vis,vis_mult);</div><div class="line">        <a class="code" href="../../d7/dfc/group__highgui.html#ga5afdf8410934fd099df85c75b2e0888b">namedWindow</a>(<span class="stringliteral">&quot;raw disparity&quot;</span>, <a class="code" href="../../d0/d90/group__highgui__window__flags.html#ggabf7d2c5625bc59ac130287f925557ac3acf621ace7a54954cbac01df27e47228f">WINDOW_AUTOSIZE</a>);</div><div class="line">        <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;raw disparity&quot;</span>, raw_disp_vis);</div><div class="line">        Mat filtered_disp_vis;</div><div class="line">        <a class="code" href="../../da/d17/group__ximgproc__filters.html#gae192a87924cbc134b75e01b79bb44718">getDisparityVis</a>(filtered_disp,filtered_disp_vis,vis_mult);</div><div class="line">        <a class="code" href="../../d7/dfc/group__highgui.html#ga5afdf8410934fd099df85c75b2e0888b">namedWindow</a>(<span class="stringliteral">&quot;filtered disparity&quot;</span>, <a class="code" href="../../d0/d90/group__highgui__window__flags.html#ggabf7d2c5625bc59ac130287f925557ac3acf621ace7a54954cbac01df27e47228f">WINDOW_AUTOSIZE</a>);</div><div class="line">        <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;filtered disparity&quot;</span>, filtered_disp_vis);</div><div class="line"></div><div class="line">        <span class="keywordflow">if</span>(!solved_disp.empty())</div><div class="line">        {</div><div class="line">            Mat solved_disp_vis;</div><div class="line">            <a class="code" href="../../da/d17/group__ximgproc__filters.html#gae192a87924cbc134b75e01b79bb44718">getDisparityVis</a>(solved_disp,solved_disp_vis,vis_mult);</div><div class="line">            <a class="code" href="../../d7/dfc/group__highgui.html#ga5afdf8410934fd099df85c75b2e0888b">namedWindow</a>(<span class="stringliteral">&quot;solved disparity&quot;</span>, <a class="code" href="../../d0/d90/group__highgui__window__flags.html#ggabf7d2c5625bc59ac130287f925557ac3acf621ace7a54954cbac01df27e47228f">WINDOW_AUTOSIZE</a>);</div><div class="line">            <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;solved disparity&quot;</span>, solved_disp_vis);</div><div class="line"></div><div class="line">            Mat solved_filtered_disp_vis;</div><div class="line">            <a class="code" href="../../da/d17/group__ximgproc__filters.html#gae192a87924cbc134b75e01b79bb44718">getDisparityVis</a>(solved_filtered_disp,solved_filtered_disp_vis,vis_mult);</div><div class="line">            <a class="code" href="../../d7/dfc/group__highgui.html#ga5afdf8410934fd099df85c75b2e0888b">namedWindow</a>(<span class="stringliteral">&quot;solved wls disparity&quot;</span>, <a class="code" href="../../d0/d90/group__highgui__window__flags.html#ggabf7d2c5625bc59ac130287f925557ac3acf621ace7a54954cbac01df27e47228f">WINDOW_AUTOSIZE</a>);</div><div class="line">            <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;solved wls disparity&quot;</span>, solved_filtered_disp_vis);</div><div class="line">        }</div><div class="line"></div><div class="line">        <span class="keywordflow">while</span>(1)</div><div class="line">        {</div><div class="line">            <span class="keywordtype">char</span> key = (char)<a class="code" href="../../d7/dfc/group__highgui.html#ga5628525ad33f52eab17feebcfba38bd7">waitKey</a>();</div><div class="line">            <span class="keywordflow">if</span>( key == 27 || key == <span class="charliteral">&#39;q&#39;</span> || key == <span class="charliteral">&#39;Q&#39;</span>) <span class="comment">// &#39;ESC&#39;</span></div><div class="line">                <span class="keywordflow">break</span>;</div><div class="line">        }</div></div><!-- fragment --> We use a convenience function getDisparityVis to visualize the disparity maps. The second parameter defines the contrast (all disparity values are scaled by this value in the visualization).</li>
</ol>
<h2>Results </h2>
<div class="image">
<img src="../../ambush_5_bm.png" alt="ambush_5_bm.png"/>
<div class="caption">
Result of the StereoBM</div></div>
<div class="image">
<img src="../../ambush_5_bm_with_filter.png" alt="ambush_5_bm_with_filter.png"/>
<div class="caption">
Result of the demonstrated pipeline (StereoBM on downscaled views with post-filtering)</div></div>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:40 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
